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Abstract: 

We  are  trying  to  understand  implicit  (un-represented  or  hidden)  human  intention,  which 
may  be  different  from  explicitly-represented  one.  Although  the  taxonomy  of  the  implicit 
intention  is  not  clear  yet,  we  hypothesize  that  the  implicit  intention  domain  consists  of  two 
axes,  i.e.,  the  sympathy  for  one's  represented  intention  and  the  sympathy  for  one's 
counterpart.  The  former  had  been  studied  in  the  framework  of  lie  detection,  while  the  latter 
is  the  new  interest  in  this  research.  When  the  subjects  read  statements  on  computer  screen, 
we  measured  fMRI,  EEG,  and  pupil  dilation.  Also,  the  subjects  were  asked  to  reply  as  'Yes' 
(Sympathy/ Agreement  to  the  statement)  or  'No'  (Non-sympathy/Disagreement). 

For  the  fMRI  experiments  nineteen  healthy  right-handed  Korean  subjects  (12  males  and 
7  females)  were  recruited  from  the  student  community  in  KAIST.  Experiments  were  held  with 
3T  MR  scanner  (Siemens  Magnetom  Verio,  Germany)  at  KAIST  Brain  Science  Research 
Center.  The  Sympathy  cases  have  higher  neural  activation  than  the  Non-sympathy  cases  in 
the  left  superior  frontal  gyrus  and  left  anterior  cingulated,  which  are  known  to  be  related 
with  self-knowledge.  Also,  the  Non-sympathy  cases  have  higher  neural  activation  than  the 
Sympathy  cases  at  the  left  fusiform  gyrus,  which  is  known  to  be  related  with  unfamiliar 
words  and  faces.  This  fMRI  experiments  approve  our  hypothesis  on  the  2nd  axis  of  the 
implicit  intention  space,  i.e.,  Sympathy  vs.  Non-sympathy  to  the  counterpart.  We  also 
conducted  linear  discriminant  analysis  (LDA)  using  pencil  beam  searching  and  identified 
brain  areas  with  more  than  70%  classification  for  the  Sympathy  vs.  Non-sympathy  axis. 

For  the  EEG  experiments  thirteen  subjects  (10  males  and  3  females)  were  recruited  and 
their  EEG  was  recorded  from  32-channel  BrainAmp  system  (Brain  Products  GmbFI,  Germany). 
Twenty-nine  electrodes  were  placed  on  the  scalp  according  to  the  International  10-20 
system.  One  electrode  for  recording  eye  movement  (EOG)  was  positioned  below  subject's 
right  eye.  Two  electrodes  dedicated  to  the  electrocardiogram  (ECG1  and  ECG2)  were  placed 
on  subject's  collarbones  in  both  sides.  Data  were  acquired  with  a  sampling  rate  of  500Hz, 
along  with  60Hz  notch  filtering.  We  work  on  two  different  electrode  selection  approaches, 
i.e.,  one  based  on  the  fMRI  results  (the  left  fontal  electrodes  such  as  F3  and  Fpl)  and  the 
other  with  Fisher's  linear  discriminant  analysis.  Both  ERP  and  frequency-band  analysis  are 
conducted.  We  had  also  trained  Support  Vector  Machine  (SVM)  classifiers  to  classify  single 
ERP  from  each  channel,  and  obtained  the  maximum  classification  rate  (78%)  at  the  central 
frontal  electrode,  Fz. 

In  conclusion  we  had  successfully  tested  a  hypothesis  on  the  implicit  intention  axis,  i.e., 
Sympathy/Non-sympathy  to  one's  counterpart,  with  fMRI  experiments.  Also,  we  showed  that 
SVM  classifiers  are  capable  of  classifying  single-trial  EEG  on  the  axis. 
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I  ntroduction: 

Human-machine  interface  had  been  developed  to  understand  explicitly-represented 
human  intention  such  as  speech,  facial  expression,  and  gestures.  However,  sometimes  it 
may  be  too  cumbersome  to  show  all  intention  sequences  explicitly.  Also,  there  may  be 
situations  when  people  is  reluctant  to  disclose  his  or  her  mind  in  daily  social  life.  It  may  be 
especially  true  in  Asian  and  military  societies  with  strong  vertical  hierarchy.  Therefore,  it  is 
desirable  to  understand  implicit  intention,  both  un-presented  and  hidden. 

We  are  trying  to  understand  implicit  human  intention,  which  may  be  different  from 
explicitly-represented  one.  Although  the  taxonomy  of  the  implicit  intention  is  not  clear  yet, 
we  hypothesize  that  the  implicit  intention  domain  consists  of  two  axes,  i.e.,  the  sympathy  for 
one's  represented  intention  and  the  sympathy  for  one’s  counterpart.  The  former  had  been 
studied  in  the  framework  of  lie  detection,  while  the  latter  is  the  new  interest  in  this  research. 

Experiment: 

Two  different  types  of  recording  experiments,  i.e.,  EEG  and  fMRI  experiments,  were 
conducted  with  the  same  experimental  paradigm  and  stimulus. 

We  had  measured  fMRI  and  EEG  signals  while  showing  somewhat  personally-sensitive 
sentences  to  the  subjects,  and  also  asked  to  respond  with  a  button  as  'Sympathy"  or 
"Non-Sympathy".  Seventy-four  stimulus  sentences  were  selected  for  the  experiment. 
Sentences  were  chosen  from  the  list  of  Minnesota  multiphasic  inventory  (MMPI)  which  is  one 
of  the  most  frequently  used  for  psychological  tests.  Selected  sentences  were  identified  into 
two  types;  affirmative  and  negative  sentences,  which  are  all  written  in  Korean. 

Although  the  stimulus  sentences  may  be  given  in  any  language,  the  subject-object-verb 
(SOV)  structure  of  Korean  language  allows  us  to  make  a  refinement  on  the  experiment 
paradigm.  In  the  SOV  languages  the  subject,  object,  and  verb  of  a  sentence  appear  in 
sequence,  and  Korean,  Japanese,  Hindi,  Latin  are  good  examples  of  the  SOV  languages.  An 
original  English  sentence  "I  read  a  book"  may  be  re-structured  in  Korean  as  "I  book  read".  In 
our  experiments,  only  two  existence  verbs  are  used;  to  be("$icj-"),  and  not  to  be("§Jcj-'')  at 
the  end  of  the  sentences.  Table  1  explains  the  difference  of  sentence  structures  between 
two  languages.  Translating  into  English,  all  sentences  are  in  present  perfect  tense  which  is 
asking  his/her  experience.  Due  to  the  sentence  order,  it  is  uncertain  whether  or  not  this 
statement  is  affirmative  or  negative  until  the  last  word.  Subject  of  sentence  ("I")  was 
omitted  and  unnecessary  adverbs  or  adjectives  were  also  left  out.  Also,  some  components 
which  can  imply  negative  form  such  as  "any",  "at  all",  "even  once"  etc.  in  contents  block. 
Thus,  subject  cannot  realize  the  type  of  sentences  beforehand.  Table  2  shows  example  of 
sentences  which  are  translated  into  English.  In  the  actual  experiments,  sentences  were  all 
given  in  Korean. 

Table  1 .  Comparison  between  Korean  and  English  Sentence  Structures 


Affirmative  Statements 

Standard  Korean 

#2# 

*£j  *|0| 

°l  Qf- 
>Ai-r 

Translation  into  English 

1 

something 

have  stolen 

ever 

Standard  English 

1  have  ever  stolen  things 

Negative  Statements 

Standard  Korean 

#2# 

S£!  *|0| 

CH  ql 
tdA’-r 

Translation  into  English 

1 

something 

have  stolen 

never 

Standard  English 

1  have  never  stolen  things 

Table  2  .  Example  of  Sentences  in  English 


Affirmative  Statements 


Negative  Statements 


Had  the  same  dream  over  and  over 

Done  anything  dangerous  for  the  thrill  of  it 

Been  told  that  I  walk  during  sleep 

Kept  from  stealing  something 

Lost  sleep  over  worry 

Worried  about  religion 

Had  difficulty  on  urinating 

Heard  so  well  it  bothers  to  me 

Worried  about  my  health 


Never  been  in  trouble  because  of  my  sex 
behavior 

Never  worried  over  money  and  business 

Never  had  a  fainting  spell 

Never  been  in  trouble  with  the  law 

Never  been  in  love  with  anyone 

Never  had  peculiar  experience 

Never  felt  like  swearing 

Never  felt  as  if  things  were  not  real 

Never  been  afraid  of  my  face  becoming  red 


Experimental  Paradigm 

Type  of  sentences  was  classified  according  to  the  affirmative  or  negative  sentence 
ending,  but  the  order  of  presentation  was  random.  Figure  1  shows  the  experiment  paradigm. 
Starting  with  fixation  cross,  which  is  accompanied  with  beep  sound,  it  was  used  to  inform 
that  next  sentence  will  be  shown  soon  and  make  them  pay  attention.  One  sentence  consists 
of  contents  block  and  sentence  end  block.  Each  block  is  shown  for  4  seconds.  After  a 
sentence  is  shown,  an  asterisk  is  presented  also  for  4  seconds.  Subjects  were  asked  to  push 
the  "Yes"  button  if  he  or  she  has  been  in  those  situations  or  agrees  on  those  statements. 
Otherwise  the  subjects  were  asked  to  push  the  "No”  button.  All  the  button  push  should  be 
made  while  the  asterisk  is  shown.  Subjects  were  asked  to  stay  still  and  try  not  to  blink  too 
much  during  the  task. 

Button  Press 
(Yes  or  No) 


4sec  4sec  2sec  2sec  2$ec 


Figure  1.  Experiment  Paradigm 


(l)EEG  Experiment 

Subjects 

Nine  healthy  right-handed  Korean  subjects  (6  men  and  3  women)  were  recruited  from  the 
student  community  in  KAIST.  They  are  all  KAIST  undergraduate  students,  and  voluntarily 
participated.  All  participants  did  not  have  a  history  of  psychiatric  disorder,  significant  physical 
illness,  head  injury,  neurological  disorder,  and  alcohol  or  drug  dependence.  After  complete 
explanation  of  the  study,  written  informed  consent  was  obtained  from  all  subjects.  The  study 
was  submitted  to  the  regular  review  in  the  KAIST  institutional  review  board  and  approved. 

Data  Acquisition 

Experiments  were  held  at  KAIST  Brain  Science  Research  Center  (N23).  The  EEG  was 
recorded  from  32-channel  BrainAmp  system  (Brain  Products  GmbH,  Germany)  and  32 
electrodes  of  an  EEG  cap  (BrainCap).  29  electrodes  were  placed  on  the  scalp  according  to 
the  International  10-20  system.  One  electrode  for  recording  eye  movement  (EOG)  was 
positioned  below  subject's  left  eye.  One  electrode  dedicated  to  the  electrocardiogram  (ECG) 
was  placed  on  subject's  collarbones  in  the  left  sides.  The  impedance  of  each  electrode  was 
maintained  below  lOkOhm  using  gel. 


Preprocessing, 

The  raw  EEG  signals  are  highly  contaminated  with  various  noises.  There  are  movement 
artifacts  made  by  human  such  as  eye  blink,  muscle,  or  heart  beat  as  well  as  artifacts  caused 
by  electrical  power  lines.  First,  acquired  EEG  signals  were  high-pass  filtered  with  a  cut-off 
frequency  at  1Hz  and  transition  band  width  0.2Hz  in  order  to  remove  line  noise.  Movement 
artifacts  cannot  be  eliminated  easily  because  one  artifact  affected  many  channels 
simultaneously.  Therefore,  independent  component  analysis  (ICA)  was  widely  used  to  find 
artifact-related  independent  component  [1-2].  ICA  is  a  statistical  method  that  maximizes  the 
mutual  independence  of  components.  So  ICA  enables  to  select  contaminated  independent 
component,  and  reconstruct  uncontaminated  signals.  In  this  study,  extended  ICA  in  EEGLAB 
was  used  to  extract  independent  components  [3],  and  artifact  components  were  removed. 

Feature  extraction 

EEG  oscillations  have  been  related  to  a  variety  of  functions  such  as  perception,  cognition, 
sleep,  etc.  For  a  long  time,  researchers  have  found  the  sensory  and  cognitive  processes  are 
modulated  by  synchronous  neural  activity  which  is  in  turn  induced  by  oscillations  [24],  A 
variety  of  studies  have  demonstrated  that  neural  oscillations  like  frontal  midline  theta  are 
closely  associated  with  memory  processes.  In  this  study,  features  were  extracted  from  the 
neural  oscillations  during  the  task  (reading  sentence  contents)  and  applied  to  the  neural 
network  algorithm  to  make  a  computational  model  for  implicit  intention  decoding.  To  do  so, 
5  band  powers  were  extracted  from  the  spectrogram  of  preprocessed  EEG  signal,  in  delta 
(1-4  Hz),  theta  (4-8  Hz),  alpha  (8-13  Hz),  beta  (13-30  Hz),  and  gamma  (30  -  40  Hz). 
With  short-time  Fourier  transform  (STFT),  spectrogram  of  each  single  trial  was  obtained. 

Then  in  each  spectrogram,  power  spectral  densities  were  summed  up  for  the  frequency 
range  of  each  band  and  averaged  for  time  domain  in  contents  block,  i.e.,  4  seconds.  Then  5 
each  representative  band  power  value  can  be  calculated  for  each  sentence  as  well  as  each 
channel.  Since  we  are  interested  in  the  higher  cognitive  function  of  brain,  channels  of 
interest  are  located  at  frontal  area  of  the  brain,  which  are  11  among  32  channels  around 
whole  brain.  We  selected  11  channels,  extracted  features,  and  applied  to  the  classifier.  We 
repeated  this  procedure  for  each  channel.  As  an  input  feature,  5-dimensional  feature  vector 
was  applied  to  an  input  of  the  classifier. 

Classification 

Input  feature  vectors  are  applied  into  the  classifier.  Classification  has  done  into  two  steps: 
training  phase  and  testing  phase.  In  training,  80%  of  input  samples  are  used  to  train 
classifier  with  known  labels.  After  training,  rest  of  20%  input  samples  is  applied  to  the 
pre-trained  classifier  and  predicts  the  labels  of  testing  samples.  Then  classification  accuracy 
can  be  obtained.  This  procedure  is  repeated  for  5  times  changing  the  dataset  of  training  and 
testing  for  reliable  performance  evaluation.  It  is  called  5-fold  cross-validation.  Classification 
performance  is  evaluated  for  each  channel  and  each  subject.  There  are  many  classifier  for 
pattern  classification,  we  selected  support  vector  machine  (SVM),  and  used  LIBSVM  tool  [6], 
The  radial  basis  function  (RBF)  kernel  is  the  most  popular  kernel  function  used  in  SVM 
classification.  RBF  kernel  on  two  samples  Xj  and  Xj  is  defined  as, 

=  eKP(-^T^  (!) 

but  it  is  possible  to  make  it  simple  using  parameter  Y  =  Tpr< 

fc(xi,Xj)  =  exp  (-y  ||xi  -  Xjll  }  ,  y  >  0.  (2) 

The  objective  function  with  soft  margin  also  includes  the  question  of  selecting 
appropriate  slack  parameter  C.  As  seen  in  Eq.  3,  slack  parameter,  also  called  penalty 
parameter,  C  decides  the  contribution  of  &,  which  is  the  degree  of  misclassification,  on 
the  objective  function. 

min  -WTW+  ClS_i.fi 

W.P.&S  1_L  ■ 1 


(3) 


-5 


Penalty  parameter  C  (log2)  Penalty  parameter  C  (log2) 

Figure  2  (a)  Training  accuracy  (b)  Testing  accuracy 


^-5  -3  -1  1  3  5  7  9  11  13  15  17  19 

Penalty  parameter  C  (log2) 

Figure  3  Training  and  testing  accuracy  versus  C  with  y  =  Z5  for  one  subject  at  one  channel 


It  is  conventional  to  find  optimal  kernel  parameter  y  and  penalty  parameter  C  in  training 
phase  using  grid  search  [26].  In  this  study,  we  also  find  the  optimal  y  and  C  which  make 
high  training  accuracy  and  apply  to  the  testing  data.  As  shown  in  Fig. 2(a)  and  (b),  greater  y 
value  shows  higher  training  accuracy.  Because  the  decision  boundary  becomes  tightly  fitting 
to  the  training  data  as  y  increases.  It  means  over-fitting,  which  is  not  as  what  we  want  it  to 
be.  When  y  =  2s,  training  accuracy  gradually  increases  with  C. 

In  Fig. 3,  training  accuracy  and  testing  accuracy  are  depicted  in  the  same  figure  with 
varying  C.  Testing  accuracy  behaves  similarly  to  the  training  accuracy  with  varying  C.  When 
C  =  2n ,  both  accuracies  goes  to  the  maximum.  In  this  manner,  we  find  the  optimal 
parameter  (C,y)  for  each  classification  procedure. 

(2)fMRI  Experiment 

Subjects 

Nineteen  healthy  right-handed  Korean  subjects  (11  men  and  8  women)  were  recruited 
from  the  student  community  in  KAIST.  They  are  all  KAIST  undergraduate  or  graduate 
students,  and  voluntarily  participated.  All  participants  did  not  have  a  history  of  psychiatric 
disorder,  significant  physical  illness,  head  injury,  neurological  disorder,  and  alcohol  or  drug 
dependence.  After  complete  explanation  of  the  study,  written  informed  consent  was 
obtained  from  all  subjects.  The  study  was  submitted  to  the  regular  review  in  the  KAIST 
institutional  review  board  and  approved. 


Image  Acquisition 

Experiments  were  also  held  at  KAIST  fMRI  center  (N23).  Functional  images  were 
acquired  on  a  Siemens  3  Tesla  MR  system  with  a  standard  head  coil.  The  volumes  consisted 
of  36  slices  (thickness  =  4mm,  no  gap)  covering  the  whole  brain.  FOV  =  220x220mm, 
matrix  =  64x64,  TE  =  28,  TR  =  2  (Flip  Angle  =  90)  GE  EPI  sequence  voxel  size 
3.4mmx3.4mmx4mm. 

Data  Analysis 

The  data  were  analyzed  with  statistical  parametric  mapping  software  package  (SPM8) 
(Wellcome  Department  of  Imaging  Neuroscience,  University  College  London)  running  with 
MATLAB  (Mathworks,  Natick,  MA).  The  imaging  data  were  realigned  to  correct  for  movement, 
and  normalized  to  the  standard  space  defined  by  the  Montreal  Neurological  Institute  (MNI) 
template.  Then  the  functional  images  were  subjected  to  two  different  analyses,  i.e.,  GLM 
and  multivoxel  pattern  analysis.  Among  19  subjects,  two  subjects'  data  were  excluded  for 
further  analysis  due  to  the  unbalanced  answer  (over  70%  in  one  class).  The  normalized 
images  were  then  spatially  smoothed  with  a  6-mm  Gaussian  kernel.  The  experiment  had  2 
event  conditions  (agreement  and  disagreement).  For  the  GLM  analysis  we  used  SPM,  and 
xjView  toolbox  (http://www.alivelearn.net/xiview)  was  also  used  to  visualize  the  results  and 
identify  the  region  of  activations  as  well  as  corresponding  Brodmann  areas.  Experimental 
conditions  were  then  contrasted  to  investigate  the  functional  contributions  of  regions  using 
standard  t-test  analysis. 


Figure  4:  A  spherical  cluster  centered  on  single  voxel  (vt)  were  defined  for  searchlight 
approach.  Local  spatial  pattern  surrounding  each  voxel  vt  was  extracted  for  pattern 
classification.  Decoding  accuracy  was  computed  for  single  voxel  and  then  moving  on  the 
whole  volume  of  the  brain. 

For  the  Multivariate  pattern  analysis  (MVPA)  we  used  analysis  technique  to  decode 
implicit  intention.  MVPA  enables  us  to  identify  cortical  regions  then  we  can  predict  whether 
the  subjects’  intention  was  agreement  or  disagreement  prior  to  their  conscious  decision  to 
press  button.  Data  samples  were  transformed  from  MRI  signal  intensity  to  units  of  percent 
signal  change,  calculated  relative  to  the  average  level  of  activity  for  each  voxel  across  all 
samples  within  a  given  run.  Spatial  smoothing  has  been  highly  controversial.  Some  of 
researchers  had  believed  a  MVPA  with  unsmoothed  data  could  maximize  sensitivity  and 
extract  full  information  in  the  spatial  pattern  [7-8].  Flowever,  others  claimed  that  smoothing 
is  nothing  to  do  with  classification  performance  and  even  improves  the  performance  for 
some  case  [9].  We  compared  the  classification  performance  with  unsmoothed,  4mm  FWFIM, 
and  6mm  FWHM  smoothed  patterns.  Then  we  used  a  "searchlight"  approach  which 
examines  the  information  in  local  spatial  patterns  of  brain  activity  surrounding  each  voxel  vj. 
A  spherical  cluster  is  defined  at  each  voxel  VjWith  radius  of  3  voxels  as  shown  in  the  figure  4. 
Local  spatial  patterns  for  single  voxel  v*  were  applied  to  the  classifier  as  an  input,  so 
decoding  accuracy  was  obtained  for  every  single  voxel  in  the  whole  volume.  For  every 
subject,  linear  discriminant  analysis  (LDA)  was  used  by  training  with  80%  of  feature  samples 
and  classification  performance  was  calculated  with  the  remaining  features.  We  repeated  this 
procedure  by  5-fold  cross-validation.  Average  decoding  accuracy  for  each  searchlight 
location  was  obtained  after  5-fold  cross-validation,  then  the  results  were  investigated 
especially  whether  it  is  significantly  above  the  chance. 


Results  and  Discussion: 

We  focused  on  whether  subjects  agree  (Sympathy  to  Others)  or  disagree  (Non-Sympathy 
to  Others)  on  the  statements  while  seeing  contents  block.  Before  sentence  end  which 
indicates  sentence  type  either  positive  or  negative  is  presented,  decision  that  agrees  or 
disagrees  on  those  statements  may  be  determined  in  the  contents  block.  "Yes"  or  "No"  is 
only  dependent  upon  existence  verbs.  We  observed  ERPs  in  contents  block.  If  pre-decision 
occurs,  and  neural  activities  are  different  between  agreement  and  disagreement,  ERPs  in 
contents  block  show  a  significant  difference  between  two  intentions.  Each  sentence  is 
classified  based  on  subject  answer.  If  subject  says  "Yes"  for  affirmative  sentence,  it  implies 
subject  agrees  on  the  contents.  If  subject  says  “No"  for  negative  sentence,  it  also  implies 
agreement  on  the  contents.  In  the  same  way,  if  subject  says  "Yes"  for  negative  sentence,  it 
means  subject  disagrees  on  the  contents.  Also,  saying  “No"  for  positive  sentence  means 
disagreement.  Let  us  assume  subject  sees  the  sentence  "I  have  never  been  to  Paris".  If 
he/she  actually  has  been  there  before,  subject's  answer  will  be  "No".  Explicit  answer  is 
negative  towards  the  sentence,  but  his/her  intention  towards  the  contents  is  positive.  It  is 
the  same  as  saying  "Yes"  to  the  sentence  of  "I  have  been  to  Paris".  "Having  been  to  Paris"  is 
a  common  truth  for  the  subject,  answer  can  be  either  "Yes"  or  "No"  according  to  the 
sentence  type.  Table  3  summarizes  how  to  classify  the  intention.  Now  each  condition  is 
named  to  YY,  YN,  NY,  and  NN  as  in  Table  3. 

Table  3  .  Classification  of  Intention 


1  ntention  to  the 
Contents 

Sentence  End 

Answer 

Condition 

Agreement 

Positive 

Yes 

YY 

Negative 

No 

NN 

Disagreement 

Positive 

No 

YN 

Negative 

Yes 

NY 

Table  4 :  9  Subjects'  average  recognition  accuracy  (%)  of  at  11  frontal  EEG  channels 


Channel 

name 

Recognition  rate 
(%) 

Fpl 

72.8 

Fp2 

75.4 

F7 

76.1 

F3 

Fz 

72.5 

F4 

72.9 

F8 

74.7 

FC5 

72.2 

FC1 

72.9 

FC2 

75.7 

FC6 

73.6 

Average 

73.9 

(1)  EEG  Experiments 

Our  experimental  design  has  demonstrated  and  decoded  implicit  intention  for  each  single 
trial  using  the  selective  attention  algorithm.  Recognition  rate  was  obtained  from  the 
subject-dependent  classification  at  11  frontal  EEG  channels  (Fpl,  Fp2,  F7,  F3,  Fz,  F4,  F8, 
FC5,  FC1,  FC2,  and  FC6)  with  5-fold  cross  validation  for  reliable  performance  evaluation. 
Table  4  shows  the  9  subjects'  average  recognition  rate  during  5-fold  cross  validation  at  11 
frontal  EEG  channels.  In  every  channel,  over  70%  of  classification  accuracy  is  obtained.  At 


channel  F7,  located  at  the  leftmost  position,  has  the  highest  accuracy  among  11  channels, 
channel  Fp2  and  FC1  follow  the  next.  Also,  Table  5  shows  the  5-fold  average  accuracy  (%) 
for  each  channel  in  one  subject.  This  subject  shows  over  80%  of  recognition  accuracy 
at  Fp2,  and  F4,  which  are  located  on  the  right  side  of  frontal  scalp.  These  results 
demonstrate  the  proposed  approach  works  well  in  implicit  intention  decoding  of  the 
proposed  experimental  task. 

Table  5  :  One  subject's  average  recognition  accuracy  (%)  of  at  11  frontal  EEG  channels 


Channel 

name 

Recognition  rate 
(%) 

Fpl 

73.0 

Fp2 

82.3 

F7 

78.6 

F3 

73.0 

Fz 

F4 

74.5 

F8 

81.0 

FC5 

79.7 

FC1 

71.5 

FC2 

74.4 

FC6 

78.5 

74.4 

Average 

76.5 

(2)  fMRI  Experiment 

As  defined  in  Table  3,  agreement  (Sympathy  to  Others)  condition  is  a  combination  of  YY 
and  NN.  In  the  same  way,  disagreement  condition  (Non-Sympathy  to  Others)  is  a 
combination  of  YN  and  NY.  To  ensure  relative  differences  between  activities  associated  with 
agreement  and  disagreement,  random  effect  analysis  of  agreement  minus  disagreement 
were  conducted.  As  illustrated  in  Figure  5,  the  relative  activations  between  agreement  and 
disagreement  are  mostly  shown  in  BA9,  dorsolateral  prefrontal  cortex,  and  BS24,  anterior 
cingulate  cortex.  BA9  is  involved  in  the  motor  planning,  organization,  regulation,  working 
memory,  etc.  BA24  is  related  to  the  rational  cognitive  functions  such  as  reward  anticipation, 
decision-making,  empathy,  and  emotion.  Flowever,  disagreement  minus  agreement  showed 
BA37  which  is  involved  word  recognition.  These  findings  are  showing  that  only  when 
agreeing  towards  the  sentence  contents  (before  seeing  the  end  of  the  sentences)  brain 
activities  related  to  the  cognitive  functions  increased. 


Figure  5.  Agreement  >  disagreement  contrast.  DLPFC  and  ACC  activities  are  shown. 


YY-(YNPNY,  NN) 


YN-(YY,NYrNN) 


Figure  6.  No  Significant  Activation  in  Sentence  End  Block. 


Figure  7.  Activated  Brain  Region  for  each  block  in  each  condition 


If  we  observe  the  4  kinds  of  conditions  in  sentence  end  block,  comparison  to  fixation 
showed  motor  and  language  related  brain  activities  dominantly.  Relative  differences  between 
conditions  did  not  show  anything.  These  results  mean  there  is  no  cognitive  process  in 
sentence  end  block,  only  associated  with  button-press  task.  Table  6  summarizes  activations 
in  this  task. 

In  this  study,  whether  subject  tells  a  truth  or  lie  is  not  our  concern.  While 
decision-making  related  to  the  personal  issue,  his  real  intention  may  be  revealed  in  EEG 
recordings.  Thus,  we  examined  the  relationship  of  ERPs  at  the  frontal  sites  and  implicit 
intention.  We  defined  implicit  intention  is  an  intention  which  can  be  observed  in  neural 
activities,  but  not  explicitly  found  yet.  In  decision  making  of  agreement  and  disagreement 
task,  we  found  intention  is  generated  in  a  brain  before  making  an  answer  explicitly.  We  have 
also  shown  that  ERP  at  Fz  can  be  used  to  predict  subject's  answer  quite  accurately  in 
advance.  Usually  ERP  analysis  has  been  done  by  averaging  all  trials  and  all  subjects  in  order 
to  average  out  the  random  EEG  noises  and  stress  out  the  common  activities  in  the  same 
class.  Flowever,  this  study  shows  that  it  is  possible  to  classify  every  single  trial.  In  other 
words,  it  is  not  necessary  to  repeat  many  trials  for  each  condition  to  find  a  common  aspect. 
Thus,  we  can  say  this  approach  is  applicable  to  the  more  natural  situation. 


Table  6  .  Summary  of  Brain  Activation 


Block 

Condition 

Brain  Region 

Description 

Contents 

Block 

YYNN-FIX 

(Agreement 

-Fix) 

Thalamus  (Left) 

kind  of  switchboard  of  information 

BA  9,  BA  32 

integration  of  sensory  and  mnemonic 
information  and  the  regulation  of  intellectual 
function  and  action  /  rational  cognitive 
functions,  decision-making,  attention 

BA  7 

play  a  role  in  visuo-motor  coordination  (e.g., 
in  reaching  to  grasp  an  object 

YNNY-FIX 

(Disagreement 

-Fix) 

Medial  Frontal  Gyrus 

Executive  mechanism,  decision-making 

Middle  Frontal  Gyrus 

Thalamus  (Left) 

kind  of  switchboard  of  information 

YYNN-YNNY 

(Agreement- 

Disagreement) 

BA  9,  BA  32,  BA  40 

integration  of  sensory  and  mnemonic 
information  and  the  regulation  of  intellectual 
function  and  action  /  rational  cognitive 
functions,  decision-making,  attention  / 
language  perception  and  processing 

YNNY-YYNN 

(Disagreement-A 

greement) 

BA  37,  Middle 
Temporal. Gyrus 

word  recognition,  within-category 
identification  /  accessing  word  meaning 
while  reading. 

Sentence 
End  Block 

YY-FIX 

(Agreement) 

BA  40 

language  perception  and  processing 

BA  6 

primary  motor  cortex 

YN-FIX 

(Disagreement) 

BA  40 

language  perception  and  processing 

Claustrum 

Consciousness 

NY-FIX 

(Disagreement) 

Insula 

Consciousness,  perception,  self-awareness 

BA  4 

primary  motor  cortex 

BA  40 

language  perception  and  processing 

NN-FIX 

(Agreement) 

BA  4 

primary  motor  cortex 

BA  40 

language  perception  and  processing 

According  to  the  results  in  fMRI  experiments,  activated  brain  regions  were  different 
during  contents  block  and  sentence  end  block.  Figure  7  shows  the  difference  between  two 
blocks.  This  figure  is  not  a  proper  contrast  to  see  the  condition  characteristics,  but  for  the 
block  characteristics.  This  is  why  4  images  seem  almost  the  same.  Contrasts  are  given  as 
contents  minus  sentence  end  for  each  condition,  so  red  or  yellow  colored  activities  are 
associated  to  the  response  in  contents  block,  while  blue  or  green  colored  activities  are 
associated  to  the  response  in  sentence  end  block.  As  clearly  seen  in  the  figure,  affected 
brain  regions  are  different,  which  implies  each  block  caused  totally  different  processes.  In 
detail,  contents  block  is  closely  connected  to  the  cognitive  processes  in  prefrontal  and 
anterior  cingulate  areas  (BA9,  BA24,  especially  agreement  condition,  see  in  Figure  5)  as  well 
as  bigger  activation  in  visual  cortex  which  may  be  due  to  larger  number  of  words  or  letters 
compared  to  the  sentence  end  block.  Flowever,  sentence  end  block  seems  related  to  the 
button  press  action  which  activities  were  shown  in  both  sides  of  motor  cortex. 

These  findings  gave  us  an  extended  idea  to  the  cultural  studies.  This  experimental  design 
can  be  applied  only  to  SOV  structured  language  natives.  Also  expression  of  agreement  and 
disagreement  is  different  according  to  the  nations.  Let  us  imagine  the  sentence  "I  have 
never  been  to  Paris."  is  presented  on  the  screen.  If  subject  is  a  Korean  and  he  has  been 
there  before,  "No"  is  his  proper  answer.  Flowever,  if  he  is  an  American,  then  he  will  say  "Yes". 
In  his  case,  agreement  is  no  more  combination  of  YY  and  NN,  but  YY  and  NY.  Agreement 
and  disagreement  is  only  dependent  upon  the  answer's  polarity  regardless  of  the  contents' 
polarity.  As  shown  in  Figure  8,  if  we  group  each  condition  following  English  manner,  we 
could  not  find  any  activation  commonly  associated  with  in  agreement  condition,  and 
disagreement  condition  either  except  simple  word  recognition  related  activities.  It  is  clearly 
seen  if  we  compare  with  Figure  5,  grouping  as  Korean  shows  us  crucial  responses  related  to 
the  cognitive  processes.  We  can  expect  that  it  might  be  observable  in  English-spoken 
subjects'  responses.  We  could  let  it  as  a  further  study  and  this  study  enables  us  to  find 
cultural  differences  in  implicit  intention  between  Korean-spoken  and  English-spoken  people. 


YYN  YYN  N  N  ( Agreemen  t  in  English)  YN  N  N-YYN  Y  ( Disagreement  in  Engl  Ish) 


Figure  8.  Grouping  Conditions  in  English  Way. 


The  observation  of  the  frontal  ERPs  or  fMRI  images  can  be  used  to  understand  implicit 
intention.  Flowever,  we  use  contact  measurement  in  limited  environment  during  experiment. 
There  is  no  way  to  measure  neural  activities  in  a  real  environment  such  as  moving  situation 
for  high  cognitive  tasks  up  to  now.  Fluman  computer  interface  should  understand  real 
intention  only  using  simple  measurements  in  moving  situation  eventually  like  recent  speech 
recognition  system  in  smartphone  does.  Measuring  device  should  be  smaller  and  convenient 
to  handle.  There  are  two  possible  solutions;  one  is  making  portable  devices  which  can 
measure  one's  neural  activities.  The  other  solution  is  investigating  the  relationship  between 
neural  activities  and  easily  measured  signals  (e.g.,  speech,  video,  etc.)  so  as  to  predict  the 
implicit  intention  only  using  non-contact  measurement  in  a  natural  situation.  Implicit 
intention  study  is  also  related  to  the  mind  reading,  which  enables  control  the  machine  by 
thinking.  In  a  not  too  distant  future,  mind  reading  can  also  be  a  general  aspect  of  future 
machine.  Implicit  intention  study  is  standing  at  the  start  line  of  the  mind  reading  research. 
Further  improvements  are  required  to  make  human  computer  interaction  in  a  more  natural 
way. 

We  also  incorporated  moving  pencil  beams  with  linear  discriminant  analysis,  and 
identified  discriminant  areas  for  the  Sympathy  vs.  Non-Sympathy  brain  responses.  As  shown 
in  Figure  12,  the  left  fusiform  gyrus  and  inferior  frontal  gyrus  are  identified. 


Left  fusiform  gyrus 


Inferior  Frontal  gyrus 


Figure  12.  Pencil  beam  analysis  to  identify  discriminant  brain  areas  for  the  Sympathy  vs. 
Non-Sympathy  responses 


List  of  Publications  and  Significant  Collaborations  that  resulted  from  your  AOARD 
supported  project: 

We  had  updated  our  experimental  paradigms  and  the  final  results  are  currently  under 
preparation  for  the  submission  to  wee-cited  peer-reviewed  journals.  Also,  to  stay  way  from 
"self-reproducing",  we  had  presented  papers  at  international  conferences  with  "abstract  only". 
Flowever,  we  had  presented  plenary  and  invited  talks  at  several  international  conferences. 


d)  conference  presentations  without  papers  (Abstract  Only) 

-  Soo-Young  Lee,  Suhyeon  Dong,  and  Dae-shik  Kim,  Understanding  human  implict 

intention  from  physiological  and  behavioral  data,  SPIE  Defense,  Security+Sensing, 

Conf.  8058:  Independent  Component  Analyses,  Wavelets,  Neural  Networks, 
Biosystems,  and  Nanoengineering  IX  April  25-29,  2011,  Orlando,  USA 

-  Soo-Young  Lee,  Artificial  cognitive  systems  with  active  learning  and  situation 

awareness  capabilities,  3rd  International  Conference  on  Cognitive  Neurodynamics, 
Hokkaido,  Japan,  June  2011,  Hokkaido,  Japan  (Plenary Talk) 

-  Soo-Young  Lee,  Implicit  Intention  Recognition  and  Hierarchical  Knowledge 

Development  for  Artificial  Cognitive  Systems,  17th  International  Conference  on 
Neural  Information  Processing  (I CONIP),  Shanghai,  China,  November  2011 
(Plenary  Talk) 

-  Soo-Young  Lee,  Understanding  human  implicit  intention  from  EEE  and  fMRI  data, 

NeuroBiology  and  Neuroinformatics  (NBNI),  Dec.  17-20,  2011,  Okinawa,  Japan  (By 
Invitation  Only) 

-  Soo-Young  Lee,  Active  Learning  and  Implicit  Intention  Understanding:  Two  New 

Functions  for  Artificial  Cognitive  Systems,  22nd  Italian  Workshop  on  Neural 
Networks,  Salerno,  Italy,  May  2012  (Plenary Talk) 

-  Suh-Yeon  Dong  and  Soo-Young  Lee,  Understanding  Human  Implicit  Intention  Based 

on  Frontal  Electroencephalography  (EEG),  International  Joint  Conference  on  Neural 
Networks  (IJ  CNN),  Brisbane,  Australia,  2012 

-  Suh-Yeon  Dong,  Byeong  Yeol  Kim,  CheongAn  Lee,  Hyunah  Song,  and  Soo-Young  Lee, 

Implicit  Intention  Understanding  and  Hierarchical  Knowledge  Development  for 
Artificial  Cognitive  Systems,  East-Asian  University  Workshop,  Feb.  2012,  Daejeon, 

Republic  of  Korea  (By  Invitation  Only) 

-  Suh-Yeon  Dong,  Medial  Prefrontal  Cortex  and  Self- releva  nee:  An  fMRI  Study, 

NeuroBiology  and  Neuroinformatics  (NBNI),  Nov.  21-23,  2012,  Seoul,  Korea  (By 
I  nvitation  Only) 

e)  manuscripts  in  preparation 

-  Suh-Yeon  Dong  and  Soo-Young  Lee,  Understanding  Human  Implicit  Intention  based 

on  the  Electroencephalography  (EEG) 

-  Suh-Yeon  Dong  and  Soo-Young  Lee,  Understanding  Human  Implicit  Intention  :An 
fMRI  Study 

f)  provide  a  list  any  interactions  with  industry  or  with  Air  Force  Research  Laboratory 
scientists  or  significant  collaborations  that  resulted  from  this  work 

-  Prof.  Riccardo  Monzatti,  ILUM  University,  Milan,  Italy:  Cross-cultural  study  on  implicit 

intention  for  Italian 
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